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CLAIMS AS PRESENTED AFTER PRELIMINARY AMENDMENT 



1. A formatted document creating system used for recording conversation comprising: 
a speech recognition device for recognizing speech of conversation, 

an information extracting section for extracting information to be recorded according to the results 
of speech recognition, 

a conversation record buffer for transforming into formatted sentence including the extracted 
information and recording this as contents of conversation, 

a case database for storing the above formatted sentence which may occur as the contents of 
conversation and includes many information, 

a topicality control section for deducing an undefined information on the basis of one item of 
information included in the sentence recorded in the conversation record buffer section by 
referencing cases stored in the case database, 

a multimodal interaction control section for inquiring the document creator about the undefined 
information recorded in the conversation record buffer to define the undefined information, 
a input/output control section for controlling the communication with the document creator using 
graphic, 

an input device for inputting the intention of the document creator into a computer system, and, 
a conversation record archive for storing therein the record of the defined conversation. 

2. A formatted document creating system defined in the claim 1 , wherein the input/output control 
section are to be controlled in GUI, the input device including any one or those combination of a 
character input device, a pointing device, and the output device including any one or those 
combination of a speech synthesizer, an acoustic sound producing device, and a graphic display. 

3. A formatted document creating system defined in the claim 1, wherein said topicality control 
section is constructed to store information that has been defined by the document creator in a 
conversation record buffer as a case in said case database. 

4. A formatted document creating system defined in the claim 1 , wherein said information extracting 
section is constructed to write some nominations regarding proper information contents to the 
conversation record buffer with the reasonable degree information based on statistics when the 
extracted information regarding the proper information seems to be true. 

5. A formatted document creating system defined in the claim 1, wherein said topicality control 
section is constructed to write some nominations regarding proper information to said conversation 
record buffer together with the reasonable degree information based on statistics when the deduced 
contents to be input in said conversation record buffer isn't decided only one thing. 

6. A formatted document creating system defined in the claim 1, wherein said multimodal 
interaction control section is constructed to inquire the document creator by a speech, a GUI or a 
character input and define the information of sentence recorded in the conversation record buffer. 



7. A formatted document creating system defined in the claim 5, wherein the multimodal interaction 
control section is constructed to inquire the document creator about information to be inquired in the 
conversation record buffer from the high nomination of a reasonable degree. 

8. A formatted document creating system defined in claim 1, wherein said system further includes 
a telephone set that enables to do conversation in apart place and a sender-side telephone number 
acquiring device that acquire the sender-side telephone number and record it in the conversation 
record buffer. 

9. A formatted document creating method used for recording conversation comprising: 
a speech recognition step for recognizing speech of conversation, 

an information extracting step for extracting information to be recorded according to the results of 
speech recognition, 

a writing in step for writing in the conversation record buffer for transforming into formatted 
sentence including the extracted information and recording this as contents of conversation, 
a case storing step for storing in a case database the above formatted sentence which may occur as 
the contents of conversation and includes many information, 

a deducing information set step for deducing an undefined information on the basis of one item of 
information included in the sentence recorded the conversation record buffer section by referencing 
the cases stored in the case database and set it in the above conversation record buffer, 
an information inquire renewing step for showing sentence including the undefined information 
recorded in the conversation record buffer by graphic to a document creator and inquiring the 
document creator about the undefined information recorded in the conversation record buffer to 
define the undefined information, and 

a conversation record step for storing defined conversation record as sentences. 

10. A formatted document creating method defined in the claim 9, wherein the information 
inquire renewing step is carried out through an input device that is any one or those combination 
of a speech synthesizer, an acoustic sound producing device, a character input device, a pointing 
device and a graphic display. 

1 1 . A formatted document creating method defined in the claim 9, wherein the case addition step 
is constructed to store information that is defined by the document creator in the conversation 
record buffer as a case in the case database additionally. 

12. A formatted document creating method defined in the claim 9, wherein the writing in step is 
constructed to write some nominations regarding proper information contents to the conversation 
record buffer with the reasonable information based on statistics when the extracted information 
regarding the proper information seems to be true. 

13. A formatted document creating method defined in the claim 9, wherein the deducing 
information step is constructed to write some nominations regarding proper information contents 
to the conversation record buffer with the reasonable degree information based on statistics when 
the deduced contents to be input in said conversation record buffer isn't decided only one thing. 



14. A formatted document creating method defined in the claim 9, wherein the information 
inquire renewing step is constructed to inquire a document creator by a speech, a GUI or a 
character input and define information of sentence recorded in the conversation record buffer. 

15. A formatted document creating method defined in the claim 9, wherein the deducing 
information set step is constructed to inquire the document creator about information to be 
inquired in the conversation record buffer from the high nomination of a reasonable degree. 

16. A formatted document creating method defined in the claim 9, wherein the speech 
recognition step is carried out by recognizing conversation through telephone and said method 
further includes sender-side telephone number acquiring step that acquire the sender-side 
telephone number and record it in said conversation record buffer that is mentioned above. 

17. A formatted document creating method defined in the claim 9, wherein the deducing 
information set step is constructed to refer a relevant case stored in the case database from the 
receiver-side telephone number recorded in the conversation record buffer and deduce other 
undefined information from an information of the sentence recorded in the conversation record 
buffer. 
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Specification 
Systgj3Q^md43aet^od^for creating formatted document 




Field of the Invention 

The present invention relates to documentation of the conversation such as 
formatted consultations, especially relates to the system and method for creating 
formatted document constructed by sentences of conversation on telephones at the 
same time. 

Background of the Technology 

If the contents of the conversation of consultation etc. are turned the 
document, it is useful to control user information and consultation contents. 
Conventionally, we recorded conversations by just leaving recording or making 
memo with a help. In the way of the former, we hear recordings again and turn the 
document by a help, but this requires great time and personal expenses to the work. 
On the other hand, in the latter one, there is a limit in the speed and we can't 
concentrate on the conversation for the need of making memo. Then, it is 
conceivable that we don't just record conversations but turn the document by using 
speech recognition section of computers. We can see this example in 6-253057 or 
5- 160925. However, there is not the proposal that prescribed about the method for 
creating formatted interaction document by using to be recorded according to the 
results of speech recognition. 

Also, the accuracy of turning the document by a speech recognition that 
mentioned above is not often sufficient for practical use. However turning document 
by a speech recognition is not sufficient to turn the documents, it included essential 
information to make formatted record For example, in the conversation between the 
physician and the patient on the telephone, contents of conversation to be recorded 
are the information about the condition, symptom and time, because it may be 
the consultation with regard to sick and injury. In a various kinds of service 
industry, such as travel agencies, the purpose, number of persons, time, place, the 
cost, schedule etc. of travel become the contents of the conversation. In this 
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situation, the contents of the conversation to be talked are limited in some degree, 
and there are many close relationships between the information and the others 
following one. 

On the other hand, thanks to the progress of the graphic user interface 
technology; it is general to display and input information in the computer by the 
combination of graphic display and pointing device. Furthermore, thanks to the 
progress of speech recognition and synthesizer technology it is possible to get the 
information by speech with computer system and operator. 

The purpose of this invention is to offer the system and method for creating 
formatted document that make the contents of formatted consultation etc.document 
efficiently 

Description of the Invention 

In this invention, as means to solve the above purposes, we create formatted 
interaction document by using the information in the turning document by speech 
recognition. 

Specifically, the present invention is a formatted document creating system 
used for recording conversation and comprises a speech recognition device for 
recognizing speech of conversation, an information extracting section for extracting 
information to be recorded according to the results of speech recognition, a 
conversation record buffer for sentence including the extracted information and 
recording this as contents of conversation, a case database for storing the case 
which may occur as the contents of conversation and it includes many information, 
a topicality control section for deducing an undefined information on the basis of 
one item of information included in the sentence recorded in the conversation 
record buffer section by referencing cases stored in the case database, a multimodal 
interaction control section for inquiring the document creator about the undefined 
information recorded in a conversation record buffer to define the undefined 
information, a GUI control section for controlling the communication with the 
document creator using graphic, an input device for inputting the intention of the 
document creator into a computer system and a conversation record archive for 
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storing therein the record of the defined conversation. 

We store therein a case which may occur as the content of conversation and 
includes many information. In the present invention, the case which may occur as 
the content of communication may he differ largely in some situations. However, in 
each situation, it is assumed that the contents of the conversation are limited in 
some degree. And, in each stored case, there are many information (to be recorded) 
that has important significance in recording conversation. We use these information 
and the relation in defining each information in the real conversation. In this way, 
creating the record of conversation, it is possible to show information which is right 
as the nominated information that composes formatted conversation record or has 
high probability that is right. Such as conversation record including an undefined 
information are showed on GUI which display on monitor of operator's computer by 
GUI control section. 

Up to here, computer makes it at the same time during the conversation. 
On the inquiry to the document creator about the undefined information that 
recorded in a conversation record buffer, the document creator inputs the answer in 
a computer system by using a various input device. Iri other words, if an undefined 
information is right, we define it as it is, and if it isn't right, we select others right 
nominated information and define it or input right answer for oneself and define it. 
This is the work that needs to click the undefined information that is right and 
define, and it enables to create conversation record including all essential 
information in a short time without disturbing the flow of the conversation. A 
defined conversation record in a conversation record archive and arranged to be 
able to use it anytime and stored. 

In the system for creating formatted document written in the claim 1, the 
present invention written in claim 2 is composed an input device including any one 
or those combination of a speech synthesizer, an acoustic sound producing device, a 
character input device, a pointing device, graphic display. 

If it is able to input the answer in a computer system, any kinds of input 
device are OK, But from operability and accuracy, it is desirable that it is any or 
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those combination of a speech synthesizer, an acoustic sound producing device, a 
character input device, a pointing device, and graphic display. 

In the system for creating formatted document written in the claim 1, the 
present invention written in claim 3 is composed a topicality control section store 
information that is defined by the document creator in a conversation record buffer 
as a case in a case database. The study effect obtained from the work to define 
undefined information by the document creator makes use of creating conversation 
record that be carried after it. 

In the system for creating formatted document written in the claim 1, the 
present invention written in the claim 4 is composed that in the case that 
information extracting about information in request ,an information extracting 
section writes some nominations regarding proper information contents to a 
conversation record buffer with the reasonable degree information based on 
statistics. When an information extracting section extracts information to be 
recorded from the results of speech recognition, it is turned documents as the 
defined information in the case that it is possible to be specified with the probability 
of 100%. In other words, in others case information extracting about information in 
request, an information extracting section writes some nominations regarding 
proper information contents to a conversation record buffer with the reasonable 
degree information based on statistics. In defining the undefined information, the 
document creator refers to this reasonable degree information. 

In the system for creating formatted document written in the claim 1, the 
present invention written in the claim 5 is composed that in the case deducing of 
the contents to be input in a conversation record buffer isn't decided only one thing, 
a topicality control section writes some nominations regarding proper information 
contents to a conversation record buffer with the reasonable degree information 
based on statistics. 

As mentioned above, a topicality control section is composed to write some 
nominations regarding proper information contents to a conversation record buffer 
with the reasonable degree information contents to a conversation record buffer 
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with the reasonable degree information based on statistics. In the case deducing of 
the contents to be input in a conversation record buffer isn't decided only one thing, 
a topicality control section writes some nominations regarding proper information 
contents to a conversation regard buffer with the reasonable degree information 
contents to a conversation record buffer with the reasonable degree information 
based on statistics. In this way, by using the reasonable degree information based 
on statistics decide the order of the contents of information that a user should select 
or narrow down it from information that in the results of turning the document by a 
speech recognition and information that a user inputs or decide. This enables to 
create the formatted contents of information efficiently. Inquiring the document 
creator about such as undefined information that was recorded in a conversation 
record buffer, define it. To the inquiry to the document creator about undefined 
information, a document creator inputs the answer into computer by using various 
input device. In other words, in the case there is a right answer ixx undefined 
information, select it and define, in the case there isn't, input right answer for 
oneself and define it. In doing these work, a document creator refers this reasonable 
degree information. 

In the system for creating formatted document written in the claim 1, the 
present invention written in the claim 6 is composed that a multimodal interaction 
control section inquire a document creator by a speech, a GUI ot a character input 
and define information of sentence recorded in a conversation record buffer By 
using means that is easy for the operator, define the information of sentence. 
Operability is more improved. 

In the system for creating formatted document written in the claim 5, the 
present invention written in the claim 7 is composed A multimodal interaction 
control section inquire the document creator about information to be inquired in a 
conversation record buffer from the high nomination of a reasonable degree. It 
makes time to reach the right information short by inquiring to the document 
creator from nomination that has high possibility for the right answer. 

In the system for creating formatted document written in the any of 
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claim 1-7, furthermore the present invention written in the claim8 includes a 
telephone set that enables to do conversation in apart place and a sender-side 
telephone number acquiring device that acquire the sender-side telephone 
number and record it in a conversation record buffer. A speech recognition section 
also can recognize a conversation speech through the telephone. A speech 
recognition section recognizes a speech of sender or receiver and then an 
information extracting section extracts information to be recorded from the results 
of speech recognition, A sender-side telephone number acquiring device gets a 
receiver-side telephone number and records it in a conversation record buffer. A 
receiver-side telephone number acquiring device gets a sender-side telephone 
number and records it in a conversation record buffer. It is possible to get some 
information about the speaker of telephone terminal and deduce others undefined 
information from the sentence information recorded in a conversation record buffer 
easily. 

The second aspect of the present invention is a formatted document 
creating method used for recording conversation and comprises a speech 
recognition step for recognizing speech of conversation, an ijoformation extracting 
step for extracting information to be recorded according to the results of speech 
recognition, a writing in step for writing in a conversation record buffer for 
sentence including the extracted information and recording this as contents of 
conversation, a case storing step for storing in a case database the case which may 
occur as the contents of conversation and it includes many information, a deducing 
information set step for deducing an undefined information on the basis of one item 
of information included in the sentence recorded the conversation record buffer 
section by referencing cases stored in the case database and set it in an above 
conversation record buffer, an information inquire renewing step for showing 
sentence including undefined information recorded in a conversation record buffer 
by graphic to the document creator and inquiring the document creator about the 
undefined information recorded in a conversation record buffer to define the 
undefined information, and a conversation record step for storing defined 
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conversation record as sentences. 

In the method for creating a formatted document written in the claim 9, 
the present invention in the claim 10 is composed that an information inquire 
renewing step including input device that any one or those combination of a speech 
synthesizer, an acoustic sound producing device, a character input device, a 
pointing device, a graphic display. 

In the method for creating a formatted document written in the claim 9, 
the present invention written in the claim 11 is composed that including a case 
addition step to store information that is defined by the document creator in a 
conversation record buffer as a case in a case database additionally 

In the method for creating a formatted document written in the claim 9, 
the present invention written in the claim 12 composed that in the case that 
information extracting about information in request, a writing in step writes some 
nominations regarding proper information contents to a conversation record buffer 
with the reasonable information based on statistics. 

In the method for creating a formatted document written in the claim 9, 
the present invention written in the claim 13 is composed that in the case deducing 
of the contents to be input in conversation record buffer isn't decided only one thing, 
a deducing information step writes some nominations regarding proper information 
contents to a conversation record buffer with the reasonable degree information 
based on statistics. 

In the method for creating a formatted document written in the claim 9, 
the present invention written in the claim 14 is composed that an information 
inquire renewing step inquire a document creator by a speech, a GUI or a character 
input and define information of sentence recorded in a conversation record buffer. 

In the method for creating a formatted document written in the claim 13, 
the present invention written in the claim 15 is composed that a deducing 
information set step inquire the document creator about information to be inquired 
in a conversation record buffer from the high nomination of a reasonable degree. 

In the method for creating a formatted document written in the any of 
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claim 9~15, the present invention written in the claim 16 is that a speech 
recognition step recognizes conversation through the telephone and this also 
includes sender^side telephone number acquiring step that acquire the receiver- side 
telephone number and record in a conversation record buffer. 

In the method for creating a formatted document written in the claim 16, 
the present invention written in the claim 17is composed that a deducing 
information set step refers a case stored in a case database from receive-side 
telephone number recorded in a conversation record buffer and deduces others 
undefined information from the sentence information recoded in a conversation 
record buffer. 

Brief Explanation of the Drawings 

Figure 1 is the block diagram of an embodiment of the system for creating 
a formatted document according to the present invention. 

Figure 2 is the block diagram of an embodiment of the system for creating 
a formatted document in a conversation through the telephone according to the 
present invention. 

Figure 3 is a diagram showing the contents of conversation between the 
doctor and the physicians. 

Figure 4 is the diagram showing the contents of the conversation record. 
The Best Mode for Carrying Out the Invention 

This is a detailed explanation of the present invention that refers to 
desirable indicated an embodiment. 

Figure 1 is a block figure that shows the entire constitution of an 
embodiment for creating a formatted document based on speech recognition of 
conversation in the present invention- As mentioned in the figure 1, the system for 
creating a formatted document based on speech recognition of conversation in this 
an embodiment comprises micro phone 1, speech recognition device 2, information 
extracting section 3, topicality control device 4, database 5, multimodal interaction 
control section 6 t conversation record buffer 7, speech synthesizer 8, acoustic sound 
producing device 9,character input device 10, GUI control section 11, pointing 
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device 12, graphic display 13 and conversation record archive 14. 

A speech recognition device 2 interprets a speech signal from a microphone 
1 as a language speech and outputs character line or results of morpheme analysis 
(the text that is disassembled every word and add information such as the parts of 
speech) In this case, a speech recognition device 2 turned the document as defined 
information and writes it to a conversation record buffer as it is, in the case it is 
possible to be specified a speech signal as information with the probability of 100%. 
In other words, in others case information extracting about information in request, 
a speech recognition device 2 writes some nomination regarding proper information 
contents to a conversation record buffer with the reasonable degree information 
based on statistics. In defining the undefined information, the document creator 
can make more appropriate judgment by referring this reasonable degree 
information. 

Furthermore, if a recording device and communication line are inserted 
between the input to a speech recognition device 2 and a micro phone 1, it isn't 
necessary to do a speech recognition in a place an time. 

And also, a multimodal interaction control section 6 collects information 
from the document creator, a speech recognition device 2 transforms information 
that the document creator offered by a speech into character line and send to a 
multimodal interaction control section 6. 

An information extracting section 3 regards rewrite mode output of a speech 
recognition device 2 as an input and extracts information to be recorded and writes 
to a conversation record buffer 7. An information to be recorded, namely 
information to be stored as information should select beforehand depends on the 
kinds information that is assumed. For example, in the case a conversation at the 
counter of travel agency is assumed, a conversation including the purpose, number 
of people, name, contact place, travel schedule, the destination, charge, lodging 
institution, way of payment etc is the information to be stored. 

As for "the purpose of the travel", in the case there are words (information) 
such as a honeymoon, a recreation of companies, a graduation trip, a private trip, a 
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group trip, a business trip, an inspection trip, a parent and child trip, summering, 
swimming, skiing, golf, scuba diving, theatergoing, watching, participation to an 
event, an information extracting section 3 extracts this as " information to be stored 
as a record" and records in a conversation record buffer. As for " number of people", 
combines words that represents number with "ninn (Unit for number of people)" 
and in the case it includes the words that represents two person such as "a couple" 
and" a newly married couple", an information extracting section 3 extracts this as 
"information to be stored as a record" and records in a conversation record buffer 7 
A topicality control section 4 deduces the contents to be input as others 
information over referring a database 5 from the information showed in a 
conversation record of a conversation record buffer 7 and writes to a conversation 
record buffer 7. This time in the case a deducing isn't decided only one thing, a 
topicality control section writes many nominations about the information with 
reasonable degree value in a conversation record buffer 7. The reasonable degree 
value is calculated based on the information written in a database 5, using the 
method of "Beise". A case database 5 stores cases to deduce the contents of others 
information from the information that was given in a conversation record buffer. In 
practice, the relationship of contents of each information is stored as a case about 
information that is defined by the document creator in a conversation record buffer. 
With the example in the counter of the travel agency that is mentioned before, a 
case is many stored " the travel schedule" is three days -two weeks, u a number of 
people" is two, "the destination" is Hawaii, the Mediterranean, west coast of the 
United States, mainly travel abroad, "the airline company " is Japanese company 
such as JAL or ANA and use business class and stay at the ocean-view suit even if 
"the cost" is a little high, "the payment" is cash and advance about information ,"a 
honeymoon". 

Under this situation, in spite of be recorded "a honeymoon", it is assumed to 
recognize that "number of people" is three, "the destination" is Atami. In this case, 
as three people contradicts under the words, "a honeymoon" and "number of people" 
a multimodal interaction control section 6 displays "two people" as first nomination 
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and "three people* as second nomination with the reasonable degree information 
and inquires to the document creator. In this case, the former reasonable degree 
point is 1.000 and the latter one is 0.000, because a number of people is "two" in all 
cases. After the document creator check it to the customer, if "two people" is right, 
the document creator clicks it and defines and "three people" is right, the document 
creator selects it and defines. For example, even if they are newly married, if there 
is a child in a couple, it may be happen to go "a honeymoon" with three people. 

Or the recognition of "a honeymoon" is actually mistake and the customer 
utters "shinjinn kennsyuu ryokou" as "shinn-kenn-ryokou" shortly in conversation, 
it may occur that a speech recognition device 2 misunderstands this as "a 
honeymoon". In this case, the document creator corrects the part of "a honeymoon" 
or "shinn-konn-ryokou" as " shinnkennryokou" by using an input device 10 of a 
keyboard etc and defines it. If it is "a new face training trip", it does not contradict 
with the destination, AtamL 

Furthermore, in the use early period stage of the system, the relationship 
between the assumed information is written by a system designer, because there 
isn't the accumulation of the case by study 

A topicality control section 4 stores information that the document creator 
defines in a conversation record buffer as a case to give a reasonable degree 
information in a database 5. A multimodal interaction control section 6 reads a 
conversation record buffer 7 and gathers information by inquiring to the document 
creator about the information that haven' t defined the contents. In this time, in 
the case the interaction by a speech is judged proper, inquiring by a speech starts. 
If it isn't proper, gathering information by a GUI or a character input device 10. 

With a request of the document creator, a multimodal interaction control 
section 6 can show the contents of a conversation record buffer 7 to the document 
creator through a graphic display 13 or a speech synthesizer 8 and an acoustic 
sound producing device. Especially, with a request of the document creator, a 
multimodal interaction control section 6 can show the turning the document 
results of a speech recognition device 2 in a conversation record buffer 7 on a 
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graphic display 13. 

In gathering interaction information from the document creator, in the 
case the choice of user's utterance is limited, a multimodal interaction control 
section 6 can make recognition of a speech more accurate by sending choices to a 
speech recognition device 2 beforehand. For example, in the case the customer 
hopes the travel to Okinawa, a multimodal interaction control section 6 sends the 
place name and its pronunciation of Okinawa to a speech recognition device 2 
beforehand. Like this, when the customer even utters the lodging, "Kunigamigunn- 
oogimimula*\ the probability that recognizes correctly is raised. 

A conversation record buffer 7 stores the contents of each formatted 
interaction information. In the case the contents aren't defined(in the case it is 
undefined information), a nomination list about the contents are given. There are a 
few cases that reasonable degree are given to each nomination. Also, a conversation 
record buffer 7 stores the turning the document results in a speech recognition 
section 2. 

A speech synthesizer 8 can transform text into speech signal about message 
to the document creator from a multimodal interaction control section 6. 

An acoustic sound producing device 9 transforms a speech signal from an 
acoustic sound producing device 8 into an acoustic. A character input device 10 of a 
keyboard transforms type input from the document creator into a signal to a 
multimodal interaction control section 6. 

A GUI control section 11 displays choices input request from a multimodal 
interaction control section 6 on a graphic display 13 and conveys choices by a 
pointing device 12 of the document creator to a multimodal interaction control 
section 6. 

A pointing device 12 conveys choices of the document creator( for example, a 
mouse click at a position) to a GUI control section. A graphic display 13 does input 
request from a mulimodal interaction control section 6 and shows contents of a 
conversation record buffer 7 based on requests of the document creator. A 
conversation record archive 14 stores the contents of a conversation record buffer 7 
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with a request of the document creator. 

In this way, the present invention can create a formatted interaction 
document by using the information in the tuning document results by a speech 
recognition* By using the reasonable degree information based on statistics, decide 
he order of the contents of information that a user should select or narrow down it 
and it enables to create the formatted contents of information efficiently. 

Figure 2 is the block diagram of an embodiment of the system for creating a 
formatted document in a conversation through the telephone that relates the 
present invention. 

A conversation or a consultation through the telephone are important 
means of communication even in the internet developed present day For example, 
in the case of the old who can't even go outside, it is difficult to learn how to operate 
computers and buy essential equipments. Then in the future, it is assumed that 
medical treatment consultation is carried out through the telephone. In that case, it 
takes it granted that doctors carry out conversation through the telephone proper 
and the contents of the consultation and prescription of medicine and others 
instructions have to be stored as a record. If the medical treatment is facetoface, it 
doesn't be a problem that doctors take time to write in a medical report in apply to 
the necessity. However, in the medical treatment through the telephone, the 
development of the system that can create the conversation record in real time, 
because the physicians tend to be anxiety if the conversation ceases or a long time. 

This erabpdiment is the one that meet such as request, basically set a 
telephone 15 instead of a microphone 1 in the system of figure 1 mentioned above 
and add a sender-side telephone number acquiring device 16 for getting a 
receiver-side telephone number and recording in a conversation record buffer 7. 

In this embodiment, a speech recognition device 2 is able to compose that it 
can switch to recognize speech that is only a sender- side, or a receiver- side, or 
both of them in the conversation through the telephone. So even if it takes to a little 
time, doctors repeat physicians' words and be able to store only doctors' words as a 
conversation records. Also in the situation, it isn't necessary to create a 
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conversation record in a real time and it includes information to be recorded in only 
the contents of the conversation of the party, it is able to compose that it does a 
speech recognize only the contents of words of the party. 

A sender-side telephone number acquiring device 16 gets a receiver-side 
telephone number and records t in a conversation record buffer 7. It is possible to 
get various kinds of information about the speaker of telephone terminal. For 
example, it is possible to get information such as name., sex, age, insurance, disease 
record, treatment record etc. and to deduce others undefined information from the 
sentence information recorded in a conversation record buffer easily. Naturally, it is 
able to composed to get the same information from a database accumulated in a 
computer of the document createrside by using the receiver-side telephone number 
get from the conversation. 
(Example l) 

Next is explanation about the system for creating a formatted document 
that is assumed the medical treatment through the telephone. At first, the operator 
writes mutual relationship between the information that is assumed in a case 
database 5. It is because there isn't the accumulation of the case by study in early 
use of period stage of the system. 

The information to be recorded are such as "temperature", '"blood pressure", 
"be anxious about", "a symptom", "when", "ache", "nausea", "the cold", "a cough", 
'"breath", "diarrhea", "where", "how", "who", "what", "an insurance", payment", 
"diagnosis". In the case a speech recognition device 2 outputs words regard to these 
words, an information extracting section 3 records these as information to be 
recorded. On the other hand, as a case about the cold, "there is a fever nearly 40 
degree and cough and a sneeze", "There isn't a fever, but pain in the joint of hands 
and legs and a face flushes", "there are the cold and snot is heavy", "there is a 
headache and also cough and sometimes yellow sputum comes out", "There is a pain 
in the throat and the back of the lung in coughing^, are accumulated in a case 
database 5. In the case it is a cold, there isn't relationship with "blood pressure" but 
with the accidents of the various bodies that comes out as "symptom" and the latter 
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relationship become high as a reasonable degree, Similar to others disease, various 
cases including many information to be recorded are accumulated. 

The consultation through the telephone between the doctors and physicians 
are recorded by using this system. Figure 3 is the contents of the conversation 
between physicians and patients. In this enforcement, transmission and receive of 
the telephone are both turned the document and recorded, 

A speech recognition device 2 outputs all transmission and receive signal of 
a telephone 5 to an information extracting section 3 after a speech recognition 
device 2 changes it into a speech signal. In the memory in an information extracting 
section 3, words that mentioned above axe registered as information to be recorded, 
and in turning documents sentence by a speech recognition device 2 of A1~A9, it 
includes information that decided to be recorded in A3-A8. On the other hand, in 
turning documents sentence by a speech recognition device 2 of Bl^B9 t it includes 
information that decided to be recoded in B2-B4, B6. In a table, underline is 
marked in information that decided to be recorded, 

An information extracting section 3 records the results of turning down the 
documents that includes information like this in a conversation record buffer 7, 
"What's happen? there is a fever and the cold and nearly 40 degree in a 
thermometer", "Is there any outstanding symptom? There is a pain in the joint and 
in the oto "How about a cough and a sneeze?", "Nothing, but there is a pain at the 
back of the lung/ 1 , "How about snot?", "Nothing", "When did you have a fever?", 
''From the last night, and there is a pain from this morning/ 1 , 11 It f s a cold. 1 ', "Please 
rest warmly, and take medicine for a fever on the market/* Here, underline is 
marked in a part that different from the conversation in real and change there bold. 

A topicality control section 4 changes "There is a pain in the throat and the 
back of the chest in coughing" that recorded as a case in a case database 5 into "a 
throat" as a nomination of " The oto " and renews the record in a conversation record 
buffer 7, and also changes a case "there is a nearly 40 deg ree fever,and a cough and 
a sneeze" into "a cough" as a nomination of " a coug h" and renews the record in a 
conversation record buffer 7. 
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A multimodal interaction control section 6 displays on a GUI of monitor 
sentences such as "What's happen?" "There is a fever and the cold and nearly 40 
degree in a thermometer", "Is there any outstanding symptom?" "There is a pain in 
the joint and in the throat" " How about a cough and a sneeze?" Nothing, but there is 
a pain at the back of the chest." "How about snot? Nothing" "When did you have a 
fever? From the last night, and there is a pain from this morning." " It's a cold." 
"Please rest warmly, and take medicine for a fever on the market" under the control 
of a GUI control section. Of course, instead of a GUI, it s possible to inquire to the 
document creator by a speech using a speech synthesizer 8 and an acoustic sound 
producing device 9. 

The document creator selects the right one about information that is 
inquired defines it by using a character input device 10 and a graphic display 13 of 
monitor combined with a pointing device 12. In the acceptable enforcement that is 
showed, a nomination of the undefined information is only one, but in the case that 
the recognition accuracy in a speech recognition device 2 is low and there are many 
nominations, it's name is similar, the nominations are more than two. Then it is 
mentioned that it is desirable to add the reasonable degree information to each 
nomination. 

Furthermore, from the defined record of the conversation, a multimodal 
interaction control section 6 writes a formatted document that includes each 
information again and records in a conversation record buffer 7. In this enforcement, 
each sentence such as "There is a nearly 40 degree fever and the cold/', "There is a 
pain in the joint and the throat.", "There is a ace in the back of the chest/ 1 , "There 
was a fever from the last night.", "There is a pain from this morning.", •It's a cold.", 
"Please rest warmly/', "Please take medicine for a cold/', are recorded as defined 
information- And this conversation record is recorded in a conversation record 
archive 14 and stored in a case database 5 as a caseCfigure 4) that includes many 
information to be recorded. Here, when the patients make a call, a sender-side 
telephone number acquiring device 16 can get information in the patients note by 
getting receiver-side telephone number. Of course, it is possible to hear from the 
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What is claimed is: 

1. A formatted document creating system used for recording conversation 
comprising, 

a speech recognition device for recognizing speech of conversation, 

an information extracting section for extracting information to be recorded 
according to the results of speech recognition, 

a conversation record buffer for sentence including the extracted 
information and recording this as contents of conversation, 

a case database for storing the case which may occur as the contents of 
conversation and includes many information, 

a topicality control section for deducing an undefined information on the 
basis of one item of information included in the sentence recorded in the 
conversation record buffer section by referencing cases stored in the case database, 

a multimodal interaction control section for inquiring the document creator 
about the undefined information recorded in the conversation record buffer to 
define the undefined information, 

a input/output control section for controlling the communication with the 
document creator using graphic, 

an input device for inputting the intention of the document creator into a 
computer system, and, 

a conversation record archive for storing therein the record of the defined 

conversation. 

2. A formatted document creating system defined in the claim 1, wherein the 
input/output control section are to be controlled in GUI, the input device including 
any one or those combination of a character input device, a pointing device, and the 
output device including any one or those combination of a speech synthesizer, an 
acoustic sound producing device, and a graphic display. 

3. A formatted document creating system defined in the claim 1, wherein said 
topicality control section is constructed to store information that has been defined 
by the document creator in a conversation record buffer as a case in said case 
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database. 

4. A formatted document creating system defined in the claim 1, wherein said 
information extracting section is constructed to write some nominations regarding 
proper information contents to the conversation record buffer with the reasonable 
degree information based on statistics when the extracted information regarding 
the proper information seems to be true. 

5. A formatted document creating system defined in the claim 1, wherein said 
topicality control section is constructed to write some nominations regarding proper 
information to said conversation record buffer together with the reasonable degree 
information based on statistics when the deduced contents to be input in said 
conversation record buffer isn't decided only one thing. 

6. A formatted document creating system defined in the claim 1, wherein said 
multimodal interaction control section is constructed to inquire the document 
creator by a speech, a GUI or a character input and define the information of 
sentence recorded in the conversation record buffer. 

7. . A formatted document creating system defined in the claim 5, wherein the 
multimodal interaction control section is constructed to inquire the document 
creator about information to be inquired in the conversation record buffer from the 
high nomination of a reasonable degree. 

8. A formatted document creating system defined in the any of 
claim l~7,wherein said system further includes a telephone set that enables to do 
conversation in apart place and a sender-side telephone number acquiring device 
that acquire the sender-side telephone number and record it in the conversation 
record buffer. 

9. A formatted document creating method used for recording conversation 
comprising, 

a speech recognition step for recognizing speech of conversation, 
an information extracting step for extracting information to be recorded 
according to the results of speech recognition, 

a writing in step for writing in the conversation record buffer for sentence 
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including the extracted information and recording this as contents of conversation, 
a case storing step for storing in a case database the case which may occur 
as the contents of conversation and includes many information, 

a deducing information set step for deducing an undefined information on 
the basis of one item of information included in the sentence recorded the 
conversation record buffer section by referencing the cases stored in the case 
database and set it in the above conversation record buffer, 

an information inquire renewing step for showing sentence including the 
undefined information recorded in the conversation record buffer by graphic to a 
document creator and inquiring the document creator about the undefined 
information recorded in the conversation record buffer to define the undefined 
information, and 

a conversation record step for storing defined conversation record as 
sentences, 

10. A formatted document creating method defined in the claim 9, wherein the 
information inquire renewing step is carried out through an input device that is any 
one or those combination of a speech synthesizer, an acoustic sound producing 
device, a character input device, a pointing device and a graphic display. 

11. A formatted document creating method defined in the claim 9, wherein the 
case addition step is constructed to store information that is defined by the 
document creator in the conversation record buffer as a case in the case database 
additionally. 

12. A formatted document creating method defined in the claim 9, wherein the 
writing in step is constructed to write some nominations regarding proper 
information contents to the conversation record buffer with the reasonable 
information based on statistics when the extracted information regarding the 
proper information seems to be true. 

13. A formatted document creating method defined in the claim 9 ? wherein the 
deducing information step is constructed to write some nominations regarding 
proper information contents to the conversation record buffer with the reasonable 
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degree information based on statistics when the deduced contents to be input in 
said conversation record buffer isn't decided only one thing. 

14. A formatted document creating method defined in the claim 9, wherein the 
information inquire renewing step is constructed to inquire a document creator by a 
speech, a GUI or a character input and define information of sentence recorded in 
the conversation record buffer. 

15. A formatted document creating method defined in the claim 9, wherein the 
deducing information set step is constructed to inquire the document creator about 
information to be inquired in the conversation record buffer from the high 
nomination of a reasonable degree. 

16. A formatted document creating method defined in the claim 9, wherein in 
the any of claim 9~15, wherein the speech recognition step is carried out by 
recognizing conversation through telephone and said method further includes 
sender-side telephone number acquiring step that acquire the sender-side 
telephone number and record it in said conversation record buffer that is mentioned 
above. 

17. A formatted document creating method defined in the claim 9, wherein the 
deducing information set step is constructed to refer a relevant case stored in the 
case database from the receiver-side telephone number recorded in the conversation 
record buffer and deduce other undefined information from an information of the 
sentence recoded in the conversation record buffer. 
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Abstract 

A formatted document creating system used for recording conversation 
comprising, a speech recognition device, an information extracting section for 
extracting information to be recorded according to the results of speech recognition, 
a conversation record buffer for recording formatted sentence including the 
extracted information, a case database for storing the above formatted sentence 
which may occur as the contents of conversation, a topicality control section for 
deducing an undefined information on the basis of one item of information included 
in the sentence recorded in the conversation record buffer section by referencing 
cases stored in the case database, a multimodal interaction control section for 
inquiring the document creator about the undefined information recorded in the 
conversation record buffer to define the undefined information, a input/output 
control section for controlling the communication with the document creator, an 
input device for inputting the intention of the document creator into a computer 
system, and, a conversation record archive for storing therein the record of the 
defined conversation. 

This system may apply to conversation through telephone. 
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Figure 1 
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Figure 2 
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Figure 3 Conversation between the physicians and the patients 



The physicians 


The patients 


A1 : Hello, this is Nishimori, 
Nishimori hospital . 






B1 : I am Sato 


A2 : What' s happen? 






B2: There is a nearly 
w 40"deareein a thermometer and 

the cold. 


A3: Is there anything 
outstanding symptoms? 






B3: There is a oain in the ioint 
and the throat. 


A4 " How about $ couqh or a sneeze? 






B4 : Nothing, but there is a pain in 
the hank of the lunq. 


A5 : How about nausea? 






B5 : Nothing. 


A6 : When did you have a cold? 






Rfi : Tt is from the, last niaht and a 
pain of the ioint is from this 
mo.rn.infl. 


A7 : It' s a cold. 




A8 : Pleas© rest warmlv and take 

mfidifvine for. a cold on the market, 






B7 : OK 


A9 : Then, take care. 
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Figure 4 the contents of the conversation record 



The name of the patient Ichiro 
Sato 47 year old male 


tel,:03-3593-6168 

add. : 1-12-14, tranomonn, 

minoto-ku, tokyo 

work : patent attorney 

card : AMEX376 1 -1 97957-800 1 8 

social insurance number : 

PXY1987-3478 

disease record : 11.20.1998 
a cold 

treatment record : 3days to stay 
hospi tal 


Symptom 


"There is a 40 degree fever .and 
the cold." 

"There is a pain in the joint 
and the throat JJ 

"There is a pain in the back of 
the lung. 33 

"There is a fever from the last 

night." 

"The pain in the joint is from 
this morning." 


Di agnosis 


"A cold" 


Treatment 


"Please rest warmly" 
"Please take medicine for a 
cold" 
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imprisonment, or both, under Section 1001 of Title 18 of the United States Code, and that such willful 
false statements may jeopardize the validity of the application or any patent issued "thereon. 

*_>a A , sToa^I INVENTOR (FIRST NAMED): 

\^0 U Name; Xadamitsu R_?U 



Residence/Post Office Address: Y 
Asia Height 20X 5-5 31 ^ 

Senzoku 3 -choree Taitou- ku 

TsJ^ya 111-Q031 Japar^ 
Citizen of; Japan 
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